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WWC Review of the Report “Looking Beyond Enrollment: 
The Causal Effect of Need-Based Grants on College Access, 

Persistence, and Graduation ” 1 

The findings from this review do not reflect the full body of research evidence 
on need-based grants for higher education. 


What is this study about? 

The study examined whether eligibility for the Florida 
Student Access Grant (FSAG), a need-based grant 
for low-income students in Florida, affects college 
enrollment, credit accumulation, persistence over 
time in college, and, eventually, graduation from 
public 2-year or 4-year institutions or private colleges 
in the state of Florida. 

The sample for this study included over 45,000 
seniors in Florida public high schools in 1999-2000 
who submitted a Free Application for Federal Student 
Aid (FAFSA). Students whose family’s ability to pay 
(known as Estimated Family Contribution) was less 
than $1 ,590 were eligible for the $1 ,300 Florida 
Student Access Grant. Students whose Estimated 
Family Contribution was greater than $1 ,590 were 
not eligible for the grant. 

Intervention and comparison groups were selected 
from the larger sample and created within narrow 
income bandwidths on either side of the $1 ,590 
eligibility cutoff for the grant. For most comparisons, 
a bandwidth of +/- $1 ,000 around the cutoff was 
used. That is, students whose family’s ability to 
pay was between $590 and $1 ,590 comprised the 
intervention group, and students whose expected 
family contribution was between $1 ,590 and $2,590 
comprised the comparison group. 


Features of Florida Student Access Grants 


The Florida Student Access Grant (FSAG) program 
is a need-based grant program for low-income 
students in Florida. Students who qualify receive a 
grant of $1 ,300, an amount sufficient to cover 57% 
of the average cost of tuition and fees for 1 year at 
a public 4-year university in Florida. 

Students in Florida may qualify for federal and 
state grants based on their families’ ability to pay 
for college. Families must complete the Free 
Application for Federal Student Aid (FAFSA), which 
is used to determine each family’s Estimated Family 
Contribution (or ability to pay). Students may then 
qualify for federal financial aid like the Pell Grant 
and state need-based grants like the Florida Student 
Access Grant. 

The study examined the impact of eligibility for the 
Florida Student Access Grant on several outcomes, 
including any college enrollment in the 7 years 
after high school, college credit accumulation up 
to 4 years after high school, associate degree attain- 
ment up to 5 years after high school, and bachelor’s 
degree attainment up to 7 years after high school. 2 
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What did the study find? 

The study authors reported that eligibility for the 
Florida Student Access Grant (FSAG) did not 
significantly increase the likelihood of ever enrolling 
in postsecondary institutions in Florida up to 
7 years after high school graduation (enrollment rates 
were approximately 81 % for grant-eligible students 
and 79% for students who were not eligible for the 
grant). However, grant-eligible students had accu- 
mulated statistically significantly more college-level 
credits 4 years after high school (approximately 
47 credits) than students who were not eligible for 
the grant (approximately 43 credits). The authors 
reported that grant eligibility did not increase the 
probability that students earned an associate degree 
within 5 years (approximately 17% in both groups), 
but did have a statistically significant positive impact 
on earning a bachelor’s degree within seven years 
after high school (increasing the rate from approxi- 
mately 24% in the non-grant eligible group to 28% 
in the grant-eligible group). 3 These findings were 
confirmed by the WWC. 


WWC Rating 


The research described in 
this report meets WWC pilot 
regression discontinuity 
standards without reservations 

This study employs a well-executed regression 
discontinuity design. 

This study focuses on a narrow subgroup of low 
socioeconomic status high school students from 
Florida who submitted Free Applications for Federal 
Student Aid. Student outcomes were only measured 
at public 2-year or 4-year institutions or private 
colleges in the state of Florida. The study, therefore, 
only includes a subset of college-bound seniors 
and may not be generalizable to students at other 
income levels or those from other states. However, 
the findings from this study may be relevant in other 
contexts to the degree that students from other 
states are demographically or socioeconomically 
similar to students from Florida. 
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Appendix A: Study details 

Castleman, B. L., & Long, B. T. (2013). Looking beyond enrollment: The causal effect of need-based 
grants on college access, persistence, and graduation (NBER Working Paper 19306). Cambridge, 
MA: National Bureau of Economic Research. Retrieved from http://www.nber.org 


Setting 

The study used administrative records from the state of Florida to evaluate the impact of 


eligibility for the Florida Student Access Grant. All study participants were high school seniors 
attending Florida high schools at the beginning of the study. 

Study sample 

Subjects were 45,727 high school seniors during the 1999-2000 school year who submitted a 
Free Application for Federal Student Aid (FAFSA). Analytic samples were created within narrow 
income bandwidths on either side of the $1 ,590 Expected Family Contribution eligibility cutoff 
for the Florida Student Access Grant. Depending on bandwidths, sample sizes varied between 
6,917 (+/- $1,000 bandwidth) and 9,501 (+/- $1 ,400 bandwidth) students. On average, students 
in the primary analytic sample of 6,917 were 60% female and averaged 17.9 years of age. The 
primary sample was 45% White, 30% Black, and 21 % Hispanic, and had an average senior 
year grade point average of 2.8 on a weighted 4.5 scale. Students were from homes with an 
average parent Adjusted Gross Income of $28,035. 

Intervention 

group 

Intervention and comparison groups were selected from the larger sample and created within 
narrow income bandwidths on either side of the $1 ,590 eligibility cutoff for the grant. For most 
comparisons, a bandwidth of +/- $1 ,000 around the cutoff was used. That is, students whose 
Expected Family Contribution was between $590 and $1 ,590 comprised the intervention group. 

Comparison 

group 

Intervention and comparison groups were selected from the larger sample and created within 
narrow income bandwidths on either side of the $1 ,590 eligibility cutoff for the grant. For most 
comparisons, a bandwidth of +/- $1 ,000 around the cutoff was used. That is, students whose 
Expected Family Contribution was between $1 ,590 and $2,590 comprised the comparison group. 

Outcomes and 
measurement 

The study had four primary outcomes: 1) enrollment at any 2-year or 4-year institution in Florida 
up to 7 years after high school, 2) credit accumulation at any 2-year or 4-year institution in 
Florida up to 4 years after high school, 3) associate degree attainment at a Florida institution 
up to 5 years after high school, and 4) bachelor’s degree attainment at a Florida institution up 
to 7 years after high school. Outcome data came from the Florida Department of Education 
K-20 Data Warehouse (KDW), which maintains longitudinal student-level records from primary 
school through postsecondary study at Florida public colleges and universities. Enrollment 
data on private 4-year colleges and universities within the state of Florida were obtained from 
the Florida Resident Assistance Grant (FRAG) database. Eligible secondary outcome data 
were also obtained from the KDW and FRAG databases. Secondary outcomes included post- 
secondary enrollment, credit accumulation, and degree attainment for various spans of time 
post high school. 
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Support for 
implementation 


Reason for 
review 


Students whose family’s ability to pay for college (Estimated Family Contribution) was less 
than $1 ,590 were eligible for the $1 ,300 Florida Student Access Grant. Students whose Estimated 
Family Contribution was greater than $1 ,590 were not eligible for the grant. Institutions were 
prohibited from awarding grants to students whose families exceeded this maximum, making 
this a sharp eligibility cutoff. 

This study was identified for review by the WWC by receiving media attention. 
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Appendix B: Outcome measures for each domain 


Postsecondary enrollment 

Postsecondary enrollment 

To measure postsecondary enrollment, study authors used the Florida Department of Education K-20 Data 
Warehouse (KDW) and the Florida Resident Assistance Grant (FRAG) databases. The KDW maintains longitudinal 
student-level records from primary school through postsecondary study at Florida public colleges and universities. 
Enrollment data on private 4-year colleges and universities within the state of Florida were obtained from the 
FRAG database. Postsecondary enrollment was a binary (y/n) variable indicating whether students were ever 
enrolled in any public 2-year or 4-year institutions or private colleges in the state of Florida. The follow-up period 
varied, with outcomes being assessed immediately and 1, 2, and 7 years post-high school. 

Credit accumulation 

Credit accumulation 

To measure credit accumulation, study authors used the KDW and the FRAG databases. The KDW maintains 
longitudinal student-level records from primary school through postsecondary study at Florida public colleges 
and universities. Enrollment data on private 4-year colleges and universities within the state of Florida were 
obtained from the FRAG database. Credit accumulation was a continuous variable indicating the number of 
credits students obtained from any public 2-year or 4-year institutions or private colleges in the state of Florida. 
The follow-up period varied, with outcomes being assessed 1 semester post-high school and 1, 2, 3, and 4 
years post-high school. 

Associate degree attainment 

Associate degree attainment 

To measure associate degree attainment, study authors used the KDW and the FRAG databases. The KDW 
maintains longitudinal student-level records from primary school through postsecondary study at Florida public 
colleges and universities. Enrollment data on private 4-year colleges and universities within the state of Florida 
were obtained from the FRAG database. Attainment of an associate degree was a binary (y/n) variable indicating 
whether students obtained an associate degree from any public 2-year institutions in the state of Florida. The 
follow-up period varied, with outcomes being assessed 3, 4, and 5 years post-high school. 

Bachelor’s degree attainment 

Bachelor’s degree attainment 

To measure bachelor’s degree attainment, study authors used the KDW and the FRAG databases. The KDW 
maintains longitudinal student-level records from primary school through postsecondary study at Florida public 
colleges and universities. Enrollment data on private 4-year colleges and universities within the state of Florida 
were obtained from the FRAG database. Attainment of a bachelor's degree was a binary (y/n) variable indicating 
whether students obtained an bachelor’s degree from any public 4-year institutions or private colleges in the state 
of Florida. The follow-up period varied, with outcomes being assessed 4, 5, 6, and 7 years post-high school. 


Table Notes: Three other outcomes were reported by the authors but not included in this report: enrollment in Florida 4-year public institutions immediately after high school, 
enrollment in Florida 4-year private institutions immediately after high school, and enrollment in Florida 2-year institutions immediately after high school. For these outcomes, the 
authors dichotomized essentially ordinal variables in ways that resulted in successful outcomes (e.g., enrollment) and nonsuccessful outcomes (e.g., not enrolled) being combined 
in the reference group (e.g., for enrollment in 4-year public institutions, enrollment in private and 2-year institutions were combined with no enrollment for the comparison). 
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Appendix C: Study findings for each domain 




Mean 


WWC calculations 


Domain and Study 

outcome measure sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Postsecondary enrollment 

Postsecondary enrollment up Sample 
to 7 years post-high school defined at 

+/- $1,200 
around the 
cutoff 

8,161 

students 

81% 

79% 

2% 

0.03 

+1 

>0.10 

Domain average for postsecondary enrollment 




0.03 

+1 

Not 

statistically 

significant 

Credit accumulation 

Credit accumulation up Sample 

to 4 years post-high school defined at 

+/- $1,000 
around the 
cutoff 

6,917 

students 

47.3 

43.0 

4.4 

0.05 

+2 

<0.05 

Domain average for credit accumulation 





0.05 

+2 

Statistically 

significant 

Associate degree attainment 

Associate degree attainment Sample 

up to 5 years post-high school defined at 

+/- $1,400 
around the 
cutoff 

9,501 

students 

17% 

17% 

0% 

0.00 

0 

>0.10 

Domain average for associate degree attainment 




0.00 

0 

Not 

statistically 

significant 

Bachelor’s degree attainment 

Bachelor’s degree attainment Sample 

up to 7 years post-high school defined at 

+/-$1,000 
around the 
cutoff 

6,917 

students 

28% 

24% 

4% 

0.06 

+2 

<0.05 

Domain average for bachelor’s degree attainment 




0.06 

+2 

Statistically 

significant 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on student outcomes, representing the average change expected for all students 
who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the 
change in an average student's percentile rank that can be expected if the student is given the intervention. Some numbers may not add exactly due to rounding. 

Study Notes: The effect sizes presented here were computed from regression coefficients and robust standard errors reported in the study. The WWC estimated the mean values 
for the intervention and comparison groups shown in the table by adding or subtracting half of the regression coefficient for the intervention effect from the grand means reported 
by the authors in Table 4. The mean differences for the outcomes were reported in Table 4 of the study report. No corrections for multiple comparisons were required. The 
p-values presented here were reported in the original study and confirmed by the WWC. 
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Appendix D: Supplemental findings by domain 


Mean WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Postsecondary enrollment 

Postsecondary enrollment 
Immediately post-high school 

Sample 
defined at 
+/- $1,000 
around the 
cutoff 

6,917 

students 

63% 

59% 

3% 

0.03 

+1 

>0.10 

Postsecondary enrollment up 
to 1 year post- high school 

+/- $1,000 
around the 
cutoff 

6,917 

students 

57% 

53% 

4% 

0.04 

+2 

<0.10 

Postsecondary enrollment up 
to 2 years post- high school 

+/- $1,100 
around the 
cutoff 

7,553 

students 

48% 

46% 

3% 

0.03 

+1 

>0.10 

Credit accumulation 

Credit accumulation up to 1 
semester post-high school 

Sample 
defined at 
+/- $1,000 
around the 
cutoff 

6,917 

students 

6.2 

5.8 

0.4 

0.03 

+1 

>0.10 

Credit accumulation up to 
1 year post-high school 

+/- $1,000 
around the 
cutoff 

6,917 

students 

12.8 

11.7 

1.1 

0.05 

+2 

<0.05 

Credit accumulation up to 2 
years post-high school 

+/- $1,000 
around the 
cutoff 

6,917 

students 

25.2 

22.6 

2.7 

0.07 

+3 

<0.01 

Credit accumulation up to 3 
years post-high school 

+/- $1,000 
around the 
cutoff 

6,917 

students 

37.3 

33.5 

3.9 

0.06 

+3 

<0.01 

Associate degree attainment 

Associate degree attainment 
up to 3 years post-high school 

Sample 
defined at 
+/- $1,300 
around the 
cutoff 

8,846 

students 

9% 

9% 

0% 

0.00 

0 

>0.10 

Associate degree attainment 
up to 4 years post-high school 

+/- $1,200 
around the 
cutoff 

8,161 

students 

14% 

14% 

1% 

0.01 

0 

>0.10 
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Mean WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

Bachelor’s degree attainment 

Bachelor’s degree attainment 
up to 4 years post-high school 

Sample 
defined at 
+/- $1,200 
around the 
cutoff 

8,161 

students 

7% 

7% 

0% 

0.01 

0 

>0.10 

Bachelor’s degree attainment 
up to 5 years post-high school 

+/- $1,000 
around the 
cutoff 

6,917 

students 

18% 

14% 

3% 

0.04 

+2 

<0.10 

Bachelor’s degree attainment 
up to 6 years post-high school 

+/- $1,000 
around the 

6,917 

students 

23% 

19% 

5% 

0.06 

+2 

<0.05 


cutoff 


Table Notes: The supplemental findings presented in this table are additional findings that do not factor into the determination of the evidence rating. For mean difference, effect 
size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors the comparison group. The effect size is a 
standardized measure of the effect of an intervention on student outcomes, representing the average change expected for all students who are given the intervention (measured in 
standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in an average student's percentile rank 
that can be expected if the student is given the intervention. Some numbers may not add exactly due to rounding. 

Study Notes: The effect sizes presented here were computed from regression coefficients and robust standard errors reported in the study. The p-values presented here were reported 
in the original study. The WWC estimated the mean values for the intervention and comparison groups shown in the table by adding or subtracting half of the regression coefficient for 
the intervention effect from the grand means reported by the authors in Table 4. The mean differences for the outcomes were reported in Table 4 of the study report. A correction for 
multiple comparisons was needed and resulted in a WWC-computed critical p-value of 0.02 for enrollment through 1 year after high school; therefore, the WWC does not find the result 
to be statistically significant. A correction for multiple comparisons was needed for credit accumulation but did not affect whether any of the contrasts were found to be statistically 
significant. A correction for multiple comparisons was needed and resulted in a WWC-computed critical p-value of 0.03 for bachelor's degree attainment after 5 years and a WWC- 
computed critical p-value of 0.02 for bachelor's degree attainment after 6 years; therefore, the WWC does not find these results to be statistically significant. 
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Endnotes 

1 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from 
the authors]) to assess whether the study design meets WWC pilot regression disontinuity design standards. The review reports the 
WWC’s assessment of whether the study meets WWC pilot regression disontinuity design standards and summarizes the study find- 
ings following WWC conventions for reporting evidence on effectiveness. This study was reviewed using the Postsecondary Education 
topic area review protocol, version 2.0. A quick review of this study was released on November 8, 2013, and this report is the follow- 
up review that replaces that initial assessment. 

2 Three other outcomes were reported by the authors but are not included in this report: enrollment in Florida 4-year public institu- 
tions immediately after high school, enrollment in Florida 4-year private institutions immediately after high school, and enrollment in 
Florida 2-year institutions immediately after high school. For these outcomes, the authors dichotomized essentially ordinal variables 
in ways that resulted in successful outcomes (e.g., enrollment) and non-successful outcomes (e.g., not enrolled) being combined in 
the reference group (e.g., for enrollment in 4-year public institutions, enrollment in private and 2-year institutions were combined with 
no enrollment for the comparison). Based on the supposition that higher-achieving, high-need students might be more responsive to 
reductions in college costs, the authors also examined whether FSAG eligibility benefitted high-achieving students more than lower- 
achieving students. These results are not described in this report because high-achieving students are not identified as an eligible 
subgroup in the Postsecondary Education review protocol. 

3 The study authors used an optimal bandwidth framework for analysis. All results reported in this single study review are based on 
students falling within the optimal bandwidth around the cutoff for each outcome. 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2014, April). 

WWC review of the report: Looking beyond enrollment: The causal effect of need-based grants on 
college access, persistence, and graduation. Retrieved from http://whatworks.ed.gov 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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